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ABSTRACT 

^ The tieed for evaluation of drug abuse treatment 

programs has-been generally recognized and mandated by law since 
1976. To learn to what extent such evaluations are actually performed 
and to obtain information about those evaluations, drug abuse 
treatment programs receiving federal fun\Js in 1979 were surveyed. 
Questionnaires were sent to a random sample of programs {N=628), with 
341 responding (54%). Among the questions asked were the number of 
actual evaluations conducted, the way data were collected and 
analyzed, and the 'sources of the evaluation personnel. Responses wepe 
tabulated in terms of three categories of evaluation, i.e., 
in-treatment, follow-up, and process/cost. Results indicated that 
substantial numbers of programs receiving National Institute for Drug 
Abuse funding at the time of the survey had conducted recent 
evaluations of their programs. In-treatment evaluation was the most 
prevalent type of evaluation conducted, with follow-up evaluation 
conducted by 20% of the j^rograms surveyed. Among the seven major 
conclusions were that: (1) larger treatment . programs with greater 
re^^ttrces were more likely to carry out evaluation; (2) program staff 
involvement in the. evaluation -process was widespread; and (3) 
significant changes in program operations resulted from the various 
program evaluations. (PAS) 



************************************************** 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document.- 

********************************************************************* 



National Institute on Drug ADuse 



o 
o 



^0 



TREATMENT 

RESEARCH 
REPORT 




Assessing Treatment: 
The Conduct of Evaluation 
Within Drug Abuse 
Treatment Programs 



Frank M. Tims, Ph.D. 
National Institute on Drug Abuse 



U A OCPAJITMCffT Of EDUCATION 
^ NATIONAL INSrm/TE OF EDUCATION 

EDUCATIONAL RESOURCES INFORMATION 
J CENTER (ERIC) 

lC\s documtot hn bMn rtproductd m 

r«c«(v«d from ih« ptmatx or ot^trmtton 
ongmanog it. 
O Mmof changM hav* tmn rmd« to imorov* 
rtpfodoction quafity 



• Poffits of vi«wocop«Monc stated inth«docu< '» 
^ |fc m«otdoi>otn«e«s»anlyr»pf««frtoffic«INIE ./ 

^ position or poficy ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^Wi^B^^Mi^B 

U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES 
Public Health Sarvlce 
g Alcohol, Drug Abuse, arid MerJtal Health Admlrjistratlon 



The Treatment Research Reports and Monograph Series ard 
issued by the Treatment Reseajpch and Assessment Brandh, 
Division of Prevention and Treatment Development, National 
Institute on Drug Pbuse (NIDA). Their primary purpose is to 
^provide reports^to the drug atxjse treatment community on the 
service delivery and policy-priented findings from 
Branch-sponsored studies, innovative service delivery models 
for different client populations, innovative treatment 
management and financing techniques , and treatment outcome 
studies. 



r 



This publication is based on data collected under NIDA contract 271-78-4629. 



Ail material appearing in this report 'is in the public domain and may be 
reproduced or copied without permission from the National Institute on Drug 
Abuse or the author.* Citatioh as to source is appreciated. 



DHHS ajblication No. (A0M)82-1218 
Printed 1982 



CONTENTS 



SlMMflftY V 

INTRODUCTION 1 

Review of Related Studies • 1 

Approach 2 

RESULTS 5 

Programs Performing Evaluation 3 

" Sources of Personnel for Evaluations ^ 

Rjrposes of Evaluations ^ 

Structure of Evaluation ^ 

Dissemination of Evaluation Results 7 

Impact of Evaluation 8 

Evaluation Needs and Resources 9 

NIDA Evaluation Manuals 9 

Variables Related to Evaluation 11 

DISCUSSION 

REFERENCES * 



ill 

-4 



sumJry 

A recent survey of 341 drug abuse treatment programs which were federally 
funded disclosed that some 44 percent reported themselves as having con- 
ducted at least one program evaluation in the preceding year. To qualify as 
evaluation, the study had to have at least the following characteristics: 
(a) a definition of relevant variables; (b) clear effort at data gathering; 
(c) data 'analysis; and (d) the reporting of study results. Using this 
definition, 40 percent of the programs surveyed reportedly performed some 
sort of in-treatment evaluation, slightly less than one-fifth reportedly 
performed followup evaluation, and 7 percent reportedly performed process or 
cost evaluation. The survey data d*a not permit conclusions regarding the 
quality of these evaluations. Due to the srteirnumber of programs identi- 
fied as conducting process/cost evaluations, details of process/cost 
evaluations were omitted from this report. 

The typical in-treatment evaluation used data from the Client Oriented Data 
Acquisition Process (COOAP) aggregated for an admission sample. Thus, about 
three-quarters of the programs reporting in-treatment evaluation claimed a 
use of CODAP admission and discharge data td permit comparison of a group or 
□roaram clients at beginning and end of treatment, or to permit comparison 
of one group of a program's clients with a group of clients from another 
program or programs.- Programs typically reported the use of several outcome 
measures for in-treatment evaluation. The most commonly reported were 
enployment status, drug use during treatment, arrests during treatment, time 
in treatment, and teasons for discharge.. Each was reported by at least 80 
percent of those programs conducting in-treatment evaluations. The majority 
of programs (58 percent) conducting in-treatment evaluation made comparisons 
of client functioning before and after treatment (i.e., admission and dis-. 
ch3rge comparisons) . Other programs made comparisons of Cheir clients 
performance with other COOAP data, or compared subgroups within their own 
programs. 

The most cotrmon purposes for performing in-treatment and followjp evalu- 
ations were as supports to internal program planning and administration. 
Each of these was reported by abpot two-thirds of programs conducting 
in-treatment evaluation. The next most frequently given purposes for 
conducting in-treatment and followup evaluations were as requirements for 
funding aRd licensure," reported by 54 and 51 percent,' respectively , of 
programs conducting in-treatment evaluation, and lesser but substantial 
percentages of those reporting followup evaluations. 

The source of personnel used in evaluations varied depending on the type of 
l^lijitIon7-Some-82-peEcent of the programs reporting in-treatment evalu- 
ation utilized their own personnel for conducting the evaluation, and 
similar patterns were observed for followup evaluation. Involvement of 
sinole State agency personnel in these evaluations was reported by about 40 
percent of the programs conducting in-treatment evaluation and 18 percent 
for those reporting followup evaluation. Lesser percentages utilized 
consultants to perform their evaluations. 

Some 55 percent of the programs involved in evaluation reportedly made 
fSLl oral reports (briefings) to clinic staff, while 69 percent reportedly 
provided written reports to program staff. About 70 percent of the programs 
involved in evaluation provided written reports of evaluation findings to 
the single State agency, and 32 percent provided oral briefings to single 
State agency staff. Most programs reported some change in program func- 
tioning consequent to their evaluations. In order of the frequency with 
which they were mentioned, the aspects of programming most often reported as 
affected were: counseling practices, intake processing, outreach activities, 
and (mentioned equally often) administrative procedures and aftercare 
services. % - 

Clinic size appeared to be related to whether or not evaluation was 
performed, with evaluation more common in the larger programs. It was not 



whether receipt of evaluation manuals produced by nIOA 
had a decisive effect on the performance of evaluation. 

# 4 

When program directors were asked to indicate needs and resources related to 
e^luation, the most commonly mentioned needs were in the areas of staff 
training and technical assistance. 

Major conclusions growing out of the study were: 

. substantial numbers of programs were found to conduct some fortr of 
evaluation, with the most common type being in-treatment (i e ' 
immediate impact) evaluation; ' v^.c, 

. larger treatment programs which have greater resources appear more 
likely to carry out evaluation; 

' ■.^Mff!'"^''^*'*'/"^!^!!?"'^' "^""^ is also inportant to acceptance and 
of evaluation, appeared to be widespread among programs 
contxicting evaluation; •» k » 

* ^I^r^^f f conducting evaluation had established regular 

Channels fbr feedback to program staff and to the single state agencies; 

. significant changes in program operation as a result of evaluation were 
oTrS^Ps^' ^''ftantial -number of programs, with the mos? cS^n kl^^s 
aJd Stercare • Processing, counseling regimen, outreach. 
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ASSESSING TREATMENT: THE CXmJCT OF EVALUATION 
J[N DRUG ABUSE TREAT>€NT PROGRAMS 



Frank M. Tim$, Ph.D. 

national Institute on Drug Abuse 



II^TROOUaiON 



This report presents the firxJings of a 
survey of 341 drug abuse treatment 
Drograms which were receiving Federape^ 
funds in 1919. The purpose of this survey 
conducted in 1979 wgs to ascertain the 
extent to which drug abuse treatment 
programs funded by the National Institute 
on Drug Abuse (NIDA) performed evaluation 
and related activities duning the year 
irtmediately prior to the survey. In 
particular, information was obtainea 
regarding actual performance of such 
evaluation activities^, the sources of 
personnel used in the evaluation, the 
nature and types of data collected and 
analyzed, the purposes for which 
evaluations were conducted, the 
dissemination of evaluation results, and 
program changes which came about as a 
result of these evaluations. Information 
was also obtained on the utilization of 
NIDA evaluation materials disseminated in 
late 1977, as well as information on 
evaluation resources available to the 
programs surveyed and evaluation related 
needs as perceived by the directors of 
those programs. 

The study was undertaken to provide basic 
information about the extent to which 
in~treatment, followup, and process/cost 
evaluations had been carried out during 
the time period in guestion. These three 
types of evaluation were defined as 
follows : 

i In-treatment evaluation was defined as 
systematic utilization of client 
status data (such as drug use, 
employment', and arrests during 
treatment and status at termination) 
during the ijourse of treatment yith 
comparisons being made against some 
fixed standard or known baseline. 

• Followup evaluation involves locating 
and interviewing a sample of clients 
after they have left treatment, with 
the objectives being the assessment of 
"treatment effects" based on 
improvement of posttreatment drug use, 
employment, criminality, and/or other 
indicators of ftlihctioning in the 
conwunity. Such followup evaluations 
should! but do hot always, involve 
comparison of treated clients with a 



group of similar clients receiving no 
treatment, less^ treatment, or ^ 
different treatment. 

• Process/cost evaluation provided a 
somewhat broader category for 
responses. Process evaluation, as its 
name implies, involved the examination 
of program operations to ascertain how 
well the program functions and * 
defining the model on which it is 
based. Cost benefit evaluation 
addresses the efficiency of a program, 
- usually expressed in terms of cost or 
service relative to a given benefit, 
a.e., cost per drug free day, or cost 
per favorable treatment termination. 

The need for evaluation of drug abuse 
treatment programs has been generally 
recognized and mandated by law since 1976 
(P.L. 9A-231), with evaluation plans 
required as part of the funding 
application for programs receiving Federal 
fufxJs through NIDA. However, it has never 
been clear to what extent such evaluations 
were actually performed, nor has 
systematic information been available on 
the details of those evaluations. The 
data collected and presented in this 
report represents a first attempt to 
provide such information, although on a 
somewhat limited basis. 

Review of Related Studies 

As is noted above, evaluation of drug - 
abuse treatment' programs funded by^NlOH 
has been required since 1976. Similar 
reguirements have existed for treatment 
programs funded by the National Institute 
of Ment&l ^tealth (NIMH) and National 
Institute on Alcohol Abuse and Alcoholism 
(NIAAA), as specified in authorizing 
legislation for those Institutes. Because 
of intrinsic differences in the nature of 
program modalitiesf as well as treatment ^ 
approaches and staffing characteristics, 
differences in the approaches used to 
evaluate funded programs of the three 
Institutes are evident. 

Community Mental Health Centers (CMHCs) 
funded by NIMH have generally used process 
measures for evaluation (Windle and Woy 
1977). The importance of outcome 
evaluation has been recognized (Woy 1980;, 
although a conference on program 
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evaluation sponsored by NIMH in 1979 for 
the puipose of examining possible 
standardized outcome measures for OlHCs 
concluded that 'the state of the art had 
not yet developed to the point that such 
outcome measures were feasible (Analysis, 
Management and Planning, Inc., 1979). 
Some writers in t^e field, notably Qarlo 
(1977) as well as Brodsky and Bigelow 
(1980) have dealt with developing systems 
for monitoring mental health treatment 
outcomes, although it is recognized that 
the multidimensionality of mental health 
treatment outcomes continues to present a 
problem for such postfreatment 
evaluation. A further problem which has 
complicated evaluation for CMHCs has been 
the inherent difficulty of defining 
evaluation (which may be for research, 
accountability, or management uses) and 
differentiating it from routine data ' 
collection and reporting activities. 

The National Institute of Mental Health 
conducted a survey of the 325 federally ^ 
funded CMHCs in 1972 (Windle and Volkman 
1973) in which they sought to identify the 
kinds of evaluative activities which were 
taking place in those centers. They 
"received 181 usable questionnaires in 
response to this mail survey and, as 
expected, found that the focus Was almost 
entirely on process measures. While it is 
not entirely clear from this sdrvey how 
many CMHCs performed no evaluation 
activities, the number would not seem to 
be high given the relatively frequent 
reporting of the service process measures 
and data on evaluation. Activities in a 
large sanple of CMHCs are currently being 
analyzed. 

A national treatment program monitoring 
system with outcome evaluation capability 
W was established in 1975 by NIAAA w^th the 
ultimate objective of bringing all funded 
alcohol treatment and rehabilitation 
programs into the system (Patterson 
1975). The system could be used for 
evaluation on a national level or 
evaluation of individual programs, as well 
as periodic compilation of client-oriented 
statistical data. Since this system 
includes development of peri^odic followup 
data such as client alcohol use and 
indices of alcohol problems and social 
functioning, the system has proven useful 
for both research and program management 
(Patterson 1979). In addition to outcome 
Evaluation, this system can also be (and 
has been) used for certain kinds of 
process evaluation. In any event, the 
centralized nature of the system as it 



existed prior to 1981 stands in contrast 
to the drug abuse treatment system where 
self -evaluation by treatment programs has 
been viewed as more appropriate. 

Drug abuse treatment programs are capable 
of being assessed with either process or 
outcome measures. In recent years, a 
great deal of emphasis has been placed on 
outcome measures, and large scale studies 
have been conducted on during-treatment 
evaluation (in which behavioral measures 
such as drug use, arrests, and employment 
were used as measures of client per- 
formance while undergoing treatment) and 
posttreatment outcome measures of drug 
use, arrest, employment, and other aspects 
of client functioning in the community 
have been used to assess the lasting 
impact of drug abuse treatment. These are 
addressed in Sells (1975) and in Simpsop, 
Savage, Uoyd, and Sells (1978) and have 
been the basis of two evaluation manuals 
produced by NIDA for use by local programs 
(Qjess and Tuchfeld 1977; Johnson, Nurco, 
and Robins 1977). These two self-eval- 
uation manuals and their utilization will 
be the focus of some of the analyses in 
the present study. 

Approach . 

The data presented in this report were 
<)btained from responses to a mailout 
questionnaire which was sent to a random 
sample of 628 programs receiving treatment 
funds from NIDA during 1979 and having a 
program (static)" treatment capacity of at 
least 25. The questionnaire contained a 
large number of questions addressing 
program evaluation and related activities 
in individual treatment programs. These 
Questions included actual evaluations 
conducted (by type of evaluation), routine 
data collection activities conducted by 
programs, data collection and analyses 
conducted for evaluation, purposes of 
evaluations conducted, sources of 
evaluation personnel, evaluation needs and 
resources of individual programs, 
dissemination and utilization of 
evaluation results, and information 
^. relating^ to utilization of NIDA evaluation 
manuals. 

A total of 341 usable questionnaires were 
.returned, a response rate of 54 percent. 
While a sample obtained from the responses 
cannot be said to be random,* owing to the 
inherent bias in response to mail 
questionnaire surveys, it should be 
pointed out that the responses obtained 
represent some 30 percent of all NIDA 
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funded programs meeting the sampling 
criteria* Moreover the modality/ 
envirortnent characteristics of the 
programs responding are in approximately 
the same proportion as those in the 
universe from which the sample was drawn, 
Specifically, A6,3 percent of the programs 
responding were drug free outpatient, 35.5 
percent ^mixed** modality, 9.4 percent drug 
free residential, 5.9 percent methadone 
maintenance, and the remainder being 
either drug free day care programs or 
~drug free other." The data were also 
supplemented by site visits to a small 
number of programs (which had indicated 
that they performed evaluation) in order 
to develop more detailed information for 
use in interpreting the survey data. 

A subsanple of 100 programs which had been 
sent quest ionnaines, but had not 
responded, was randomly selected for 
telephone interviews in order to assess 
bias in the mail questionnaire sample. Ot 
these 100 programs, 82 program directors 
were interviewed, with the remaining 18 
refusing to be interviewed. Comparisons 
were made of pawillel items in the 
questionnaires of the mail response sample ^ 
and telephone interviews of the 82 program 
directors. These comparisons suggested 
that programs conducting in-treatment 
evaluation and followup evaluation tended 
to be over-represented among the mail 
questionnaire respondents, while programs 
conducting process/cost evaluations were 
under-represented. While the telephone 
interview results strongly suggest that 
the mail questionnaire sample is somewhat 
more active in evaluation and cannot be 
said to be wholly representative, the 
findings of the mail survey are important 
since they provide data on the nature and 
performance of evaluation by those service 
delivery programs making some investment 
in evaluation and reflect a comparatively 
large response (5A percent) to a mail 
questionnaire. 

RESULTS 

Responses bythi program directors were 
tabulated generally in terms of. the three 
categories of ^ e^aluation-in-treatment , 
followup, and prbcess/cost. Generally 
accepted definitions (as previously 
specified) were used in the instructions 
sent with the questionnaire, although some 
degree of interpretation by the 
' respondents was possible* ^ 

Because over-reporting based on liberal 
interpretations of the definitions 



provided was evident in the data obtained, . 
a stringent criterion was used in 
analyzing data. Programs were considered 
to have performed evaluation only if they 
(1) indicated use of a data gathering 
plan. (2) defined criterion (e.g., . 
outcome) variables, (3) analyzed data by 
means of sbme type of comparison, and W 
reported the results obtained to program 
or other staff. 

Proarams Performing Evaluation 

As table 1 shows, some percent of the 
programs surveyed were viewed as having 
performed an evaluation cXjring the 
previous 12 months, i.e., conducted 
evaluations which met the study criteria 
enumerated above. About AO percent of the 
programs conducted in-treatment 
evaluation, 19 percent followup 
evaluation, and 7 percent process /cost 
evaluation. Thus, it is evident that^ 
significant number of programs conducted 
both in-treatment and followup 
evaluation. The data obtained in this 
survey did not permit judgments regarding 
the quality of the evaluations reported. 
A small -number of site visits to programs 
claiming to have performed evaluation were 
conducted in order to obtain additional 
information on these evaluations. It 
became evident that the definition of 
evaluation varied among program 
directors. In some cases the evaluation 
performed was rigorous, and in many 
instances evaluation was considerably less 
formal. Thus, the data ^resulting from 
studies will vary in "research quality 
although not necessarily in perceived 
utility to the treatment program. 



Table 1* Percentage of programs reporting 
performance of evaluation OJring 
preceding 12 months 
(N=3A1) 



Type of evaluation 



Percent 



In-treatmeat evaluation 
Followup evaluation 
Process/cost evaluation 
Total conducting any 
evaluation 



39.9 
19. A 
6.7 

A3. 7 



NOTE: Programs were permitted to indicate 
as many as three categories of evalu- 
ation (i.e., multiple responses). 



A further issue concerns the findings with 
regard to process/cost evaluation. In 
view of the relatively small number of 
programs considered to have conducted 
process/cost evaluations, this type of 
evaluation is omitted from the more 
detailed tabulations which follow, since 
the small numbers would be subject to 
chance fluctuations and interpretation 
would be difficult. 

Sources of Personnel for Evaluations 

Program directors were asked ta identify 
the source(s) of personnel used in each 
kind of evaluation. As the responses in 
table 2 show, the great majority of these 
programs relied on their own personnel to 
carry out these evaluations, although many 
programs reported more than one source of 
personnel (multiple responses were 
permitted). Some 82 percent of the 
programs conducting in-treatment eval- 
uation-reported using their own staff; 18 
percent used consultants selected by the 
program. About 40 percent of these 
programs used single state agency (SSA) 
staff for in-treatment evaluation, while 7 
percent used consultants provided by the 



program conducting the evaluation and only 
a minority of the programs utilized SSA 
staff. Since the role of the SSA staff 
was not defined, it is unclear to what 
extent that staff acted in an advisory or 
in a directive role. Programs responding 
were, of course, permitted to report as 
many sources of personnel as they actually 
used and many programs obviously relied on 
more than one source of personnel. 

ajrposes of Evaluations 

As may be seen in table 3, the purposes 
for performing evaluations were varied, 
with many programs reporting more than one 
purpose for a given evaluation. Among the 
purposes given for in«-treatment evaluation 
both internal and external considerations 
were evident. Of these 136 programs 
reporting in-treatment evaluation, 68 
percent cited internal planning and 65 
percent program administration as purposes 
of the evaluations. Also prominent were 
funding and licensure which were cited by 
percent and 52 percent, respectively. 

Among the 66 programs reporting followup 
evaluation, 61 percent cited internal^ 
planning and 55 percent gave program 



Table 2. Source of personnel for in-treatment and followup evaluations 
reported by drug abuse, treatment programs ^ 



Source of personnel {%)- 



In-treatment 
evaluation 
(N=136) 
(percent) 



Followup 
evaluation 
(N=66) 
(percent) 



Program staff 

Program consultants 

SSA staff 

SSA consultants 

Other 



82. A 
18.A 
39.7 
6.6 
6.6 



72.7 
7.6 

18.2 
A. 5 
6.1 
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Programs conducting followup evaluation 
tended to rely primarily on their own 
staff. Of the programs conducting this 
type of evaluation, 73 percent used their 
own program staff, 8 percent used con- 
sultants selected by them, 18 percent used 
SSA staff members, and A percent used 
consultants provided ^ the SSA. 

Thus, the' personnel used in evaluations 
were primarily under the direction of the 



administration as purposes. Funding and 
licensure were also considerations being 
reported by 33 percent and 27 percent of 
these programs, respectively. 

Structure of Evaluation 

Program directors were asked to indicate 
major characteristics of the evaluations 
actually conducted. Of particular 
interest were the data sources used. 



Table 3. Purpose of in-treatment and followp evaluations reported 



Rjipose 



In-treaunent 
evaluation 
(N=136) 


FoXlovAjp 
evaluation 
(N=66) 


51.5 


27.3 


67.6 


60.6 


26.5 


21.2 


53.7 ^ 


33.3 


8.1 


6.1 


6A.7 


54.5 


23.5 


19.7 


8.8 


6.1 



Licensure 
Planning 

1) internal 

2) external (issued 
to umbrella agency) 

Funding 

Legislative response 
Program administration 
General research 
Other 



criterion variables used, and analyses 
conducted. Because the questionnaire 
requested a large volume of information, 
it wa^ necessary to address these 
questions in the most straightforward 
manner. Thus, only general outlines ot 
evaluation characteristics were obtained. 

The use of the Client Oriented Data 
Acquisition Process (COOAP), a data system^ 
which included client admission and 
termination data in all federally funded " 
drug treatment programs, was of particular 
interest. Because of the nature of CODAP, 
it was customarily collected at the clinic 
or program level and thus could be 
aggregated for use in evaluation. 
Three-quarters (76 percent and 72 percent, 
respectively) of programs conducting 
in-treatment evaluation reported use of 
COOAP admissions and termination data, 
with virtually all programs which utilized 
COOAP termination data also utilizing 
CODAP admissions data. 

The remainder used some other data system 
such as the single State agency 
information system. CQOAP admission and 
termination reports were also used by A8 
percent and 51 percent, respectively, of 
programs conducting followup evaluation. 

A wide range of criterion measures was 
reported (table A), with the most common- 
being client employment, educational 
status, drug use, arrests, time in 
treatment and reason for discharge. Each 
of these was reported by at least 72 
percent of the programs conducting 
Jln-treatment evaluation. A variety of 
drug use measures were reported, with 



programs obviously relying on data from a 
multiplicity of sources. Self-reports of 
drug use by type of drug and self-reports 
by frequency were each reportea by at 
least 72 percent of the programs 
conducting in-treatment evaluation. 
Although not shown^in table A, it was 
furth^tr noted that fully 99 percent of 
programs conducting evaluation used at 
least one of these two drug use measures. 
In addition to the self-reports, programs 
•conducting in-treatment evaluation also 
indicated reliance on staff reports of 
client drug use (50 percent) and reports 
of client drug use from medical records 
(57 percent). It is not clear just how 
much overlap exists between these 'sources 
of data. Although not shown in table A, 
65 percent of the programs conducting 
in-treatment evaluation also used 
urinalysis' to monitor client drug use 
during treatment and an additional 15 
percent used some other method to monitor 
client drug use during treatment. It is 
reasonable to expect that these staff 
reports of drug use and reports of drug 
use from medical records include the 65 
percent of programs monitoring drug use 
through urinalysis. 

Similar outcome measures were used by 
programs conducting followup evaluation. 
Employment, self -reported drug use, and 
arrests following treatment were each 
reported as outcome variables by more than 
85 percent of those programs. Lesser 
percentages reported other measures of 
productivity such as education and 
homemaker status. Days incarcerated 
following treatment were reported by 47 
percent of the programs conducting 
followup evaluation. 



Comparisons of dat^ were made in -any of 
several ways (see table 4). Some 58 
percent of those conducting in-treatment 
evaluation compared clients' pre treatment 
status on criterion variables with their 
status at termination. Comparison of , 
subpopulations wittiin clinics was reported 



by 46 percent of programs conducting * 
in-treatment evaluation, while 44 percent 
made comparisons between clinics or 
modalities. Comparisons with recently 
published CODAP data were reported by 43 
percent of those. conducting in-treatment 
evaluation* 



Table 4. Structure ^of evaluations 



In-treatment 
evaluation 
(N=136) 



' FollOWUp 

evaluation 
(N=66) 



A. OuCfcome Variables 

1. Client's st2i5ility measures 

a) employment status 

b) educational status 

c) participation in skill 

development programs 

d) ttomemsiker status 

2. Client's drug use ^ 

a) ^ielf-reporte^^ drug use 
^ , tiy d»jig tyfe 

b) self-|fepqrted cjrug use 

by frequency of use 

c) staff report ofeg;rug use ^ 

d) drug use from medical ^cords 

e) daysJiSfnjg free 

3. Client's ^criminal activity 

a) arrests during treatment 

b) arrests following treatment ^ 

c) days incarcerated 

4. Program participation 

a) reason for {Jiscb^rge 

b) time in t#atment 



84.6 
83.8 

'7^.5 
"•57.4 



^ 78.7 

72.1 
59.6 
56.6 
^4.9 



80.9 
33.1 
57.4 



B* Type of Comparisons ^ « 

1. Compared oul^e variables, in sub- 

populations within your clinic 

2. Compared outcome variables between 

clinics or modalities 

3. Compared variables with published ^ 

data 

a) recent CODAP data 
^ b) NDATUS data ' vl. 

c) tables in evaluation handbook' 

d) other ^ 

4» 'Compared variables before and 
after treatmenl/ 

5. Performed other analysis using 
statistical methodology 



80.9 
81.6' 

46.3^ 
44.1 



43.4 
11.0 
5*9 ' 
5.1 



58.1 



8.1 



68.2 
47.0 



86.4 

60.6 
51.3 



54.5 
87.9^ 
47.0 



57.6 
60.6 



48.5 



36.4 



28.8 
9.1 
1.5 
3.0 



66.7 



10.6 
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Pretreatroent to posttreatment comparison.^ 
of outcorne variables was reported by 67 
percent of programs conducting followup 
evaluation, while percent made 
conoarisonsc between subpopulations in 
their clinic, 36 percent made comparisons 
between modalities or clinics, 
percent made comparisons with publisheo 
COOAP data. 

Thus, while a number of outcome variable 
'measures were reported, there was a 
tendency for programs ta report client 
stability measures (especially • 
enployment), client drug us^, and client 
criminal behavior with relatively high 
frequency. While these were reported for 
'the two categories of evaluation, otner 
variables -aj^rrsrtime in treatment and 
'reason for treatment termination were also - 
encountered with high frequency among ^ 
those programs conducting in-treatment 
evaluation. Comparisons of variables 
pretreatment and posttreatment (or at 
dischatge) were commonly reported, both 
for in-'treatment and -followup evaluations, 
as wejre comparisons within clinics, 
betw^ clinics, and with recent Published 
COOAP data. As previ«i|sly mentioned, no 
attenpt was made to judge the c^ality of 
these evaluations. 



Dissemination of Evalu ation Results 

The data in table 5 depict the percentage- 
of programs utilising different avenues 
for dissemination of ' evaluation findings. 

. Qearly, since the program may use more 

•than one method of disseminating findings, 
multiple 'responses are reflected lo the 
data. The data obtained did not permit a 
separate breakout by type of evaluation 
ar^^therefore table 5 is ba^ed on all 149 
programs which reported conducting 
evaluation. Responding prograJns were ^ 

. asked' to indicate how evaluation findings 
were disseminated and to what audiences. 
They were also asked whether these methods 

' of dissemination were used on a regular or 
an irregular basis. Thus, percentages may 
be sunmed across .regular and irregular use 
of ajDarticular avenue of dissemination to 
a particular audienc^. 

As table 5 shows, a variety of methods 
were reported for disseminating evaluation 
results to program staff, single State 
agency staff, and to others in the field, 
and many programs reported more than one 
method. "Some 46 percent of the programs 
used formal oral reports to program staff 
on a reg^ilar basis, while 9 percent used 
* this type of presentation irregularly. 



Table 5. Method and regularity of dis^^^^on of evaluation results 



Method of dissemination 



Regularly 



Irregularly Total 



^ Treatment program staff 

1. formal oral reports 

2. written jeports 

3. informal communications 

B. Single State agencies 

1. formal oral reports 

2. " written reports 

3. informal communications 

C. National Institute on Drug Abuse 

1. formal oral reports , 

2. written reports 

3. i-nformal communications 

D. Other agencies 

^1. formal oral reports ■ 

2. written reports 

3, informal communications 

E. Professional journal article 



46.3 
57.7 
40.3 



28.2 
59.7 
19.5 



12.1 
31.5 
6.0 



8.7 
29.5 
7.4 

5.4 



t«TE: Multiple responses were permitted, as appropriate. 
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8.7 
11.4 
15,4 



3.4 
10.7 
11.4 



3.4 
8.1 
1.3 



2.0 
4.0 
1.3 

6.0 



55.0 



69 
55 



31. 
70. 
30. 



15.5 
39.6 
7.3 



10.7 
33.5 
8.7 

11.4 



Results .were communicatW to program staff 
through^writte© reports on a regular basis 
in 58 percent of the programs and. on an ^ 
irregular basis in ll^percent. 'Informal 
communication of results on a regular ' 
basis was reported by 40 percent of the 
programs, while 15 percent imported such 
conntinications on an irregular basis, 

• Communication of evaluation firxJings to 
the single State agency was reported by a 
najority of^the prdgrams' which conducted 
evaluations^- Fully 70 percent of these 
programs provided written reports to the 
single State agency; -with 60 percent doings' 
so on a regular basis and about 11 percent 
pn an irreguldr basis. Regular use of 
formal oral reports to the single State 
agency was report^ by 28 percent of the; 
programs conducting evaluation, while y 
percent did so irregularly. Informal * 
cootTMnications were risported by 31 percent 
of the programs. 

Forty percent of the programs conducting 
evaluation^ provided written .feports to the 
National Instijfeute on Djug Abuse; some 15 ' 
percent reported formal oral reports to 
NIDA, and 7. percent reported informal 
communication of results to NIOA. 

Results were also disseminated to other 
agencies through written reports in the 
case of 33 percent of the prograns 
conducting evaluation. Formal oral 
briefings and informal communication were 
reported by relatively small percentages 
of these programs. Articles published in 



the] professional journals on the basis of 
evaluation findings were 'reported by 11 
percent of the programs. 

Thus, the m^ority of programs .conducting 
evaluation tended to hkve well established 
channels for communicating 'their 
findings. As was appropriate, the prime 
recipients of this information tended to ' 
be program staff, although single State 
agencies, also received regular reports. 
Predictably, infomal communication of 
results tended to be most frequent «at the 
program level and less with increasing 
distance from the' program. Also provision 
of written reports to the single State 
agencies— which acted as a prime funding 
. source— was much more common than to any 
other agency. 

Impact of Evaluation 

Table 6 depicts the reported changes in 
program operations which werie identified 
as resulting from the evaluations. It, 
should be noted th^t, as was the case with 
the previous table, percentages werfe based 
on the 1^9 pfogramS considered to have 
conducted evaluation, and multiple 
responses were, permitted. The most, 
prevalent change was in counseling 
regimen, which was reported by 58 percent 
of the programs conducting evaluation, 
while the second most common change was in 
intake prcJcessing (50 percent), followed 
by expanded outreach efforts (A6 
percent). Changes in adminktrative 
procedures and aftercare xw^ each 



f ^^f J2^^ operations reported by drug abuse 

treatment programs as resulting from evaluation 
(N=1A9) 



Changes 



Percientage 



Staff size increased 

Funding increased 

Client contact time increased 

Operating hours extended 

Intake processing altered 

Decreased staff/client ratio 

Increased client/staff ratio 

Change? in physical environment 

Changes in counseling regimen 

Developed new modality 

Changed administrative procedures 

Expanded outreach 

Changes in aftercare provision 

Other 



NOTE: Multiple responses were permitted, as appropriate. 



20.1 
20.8 
26.2 
20.1 
A9.7 
10.7 
15.4 
26.2 
57.7 
29.5 
AO. 3 
A5.6 
AO. 3 
3. A 
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reported by AO percent of the programs. 
Some 29 percent of the programs reported 
developing a new modality or clinic as the 
result" of evaluation, while .about 
one-fourth increased client contact time. 
About one-fourth of the programs reported 
changes in the physical environment of the 
treatment setting. 

Thus, it is clear that change as a 
consequence of evaluation. was widely 
reported, and that individual programs may 
have made numerous changes subsequent to a 
given evaluation* Two major thiusts of 
program changes were improvement of 
treatment through changes in services 
provided and expansion of services in an 
effort to /reach more clients. 

Evaluation Needs and Resources 

Program directors were asked to identify 
their needs and resources for conducting 
evaluation. These two items were 
presented in op§D-ended format to allow 
the program dir^tors to identify any need 
or resources they considered salient, and 
no attempt was made to dictate response 
categories in advance. The responses were 
then categorized for computation. Table 7 
presents responses to these two items. < 
Evaluation needs were identified by some - 
57 percent of the programs, while the 
remaining A3 percent d'id not identify 
^^eds. Some A2 percent of the programs 
identified evaluation resources available 
to them. 

Among the programs surveyed, the most 
frequently reported evaluation need was 



staff training C37 percent) followed by 
data organization (11 percent). Regarding 
evaluation resources available to the 
programs, about 15 percent of the programs - 
reported trained staff, while an 
additional 15 percent reported outside 
resources (such as consultants). An 
additional 7 petcent reported access to a 
data system as an evaluation resource. 

While the relatively low response rate to 
these two items makes it difficult to 
generalize, the pattern of responses 
suggests that training of program staff in 
evaluation and assistance in data base 
organization are two common nepds. 
Outside resources such as consultants and 
data systems can be useful, although it is 
widely recognized that involvement of 
•program staff in, evaluation is important 
if those evaluations are to be accepted 
and have the greatest impact. , 

NIDA Evaluation Manuals 

A subsidiary objective of this study was \ 
to examine the extent to which evaluation' 
manuals prodqped by NIDA and disseminated 
to the field in 1977 had been received, 
retained, and were in use for evaluation 
at the time of the' sur.vey reported here. 
About 2 years had elapsed between the 
initial dissemination of tw6 evaluation ^ 
manuals — Manual For Drug Abuse Treatment 
PtOoram Self-Evaluation jUiess and 
Yir^^foiH anri lionductinQ Followup 

Research in Drug Treatment Programs 
(Johnston, Nurco, and Robins 1977;— and 
the time of the survey. The percentage of 
programs which reported having received 



Jable 7. Evaluation needs and resources reported by drug abuse 
treatment programs surveyed 
(N=3A1) 
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Item 



Pelrcent 
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B. 



Needs 
1. staff training 
'2. data organization 

3. peer review 

4. NIDA materials 

5. outside consultants 

6. none reported 


37.0 
11.4 
1.2 
2.9 
A.7 
42. 8 


Resources 

1. NIDA materials 

2. outside resources 

3. trained staff 

A. technical assistance 

5. data system 

6. none reported 


1.8 
15.0 
15.5 

2.6 
* 7.0 
58.1 
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either of these documents (which had been 
mailed to all NIOA-funded treatment 
programs) was disappointingly small* As 
table 8 shows, 62 percent of the programs 
reported having received the Manual for 
tojQ Abuse Treatment Program Self-Eval - 
uation , and 45 percent had reported 
receiving Conducting Followup Research in 
Drug Treatment Proqr^ t with 40 percent 
of the programs reporting having received 
both manuals* In fairness, it should be 
pointed out that staff turnover in drug 
abuse treatment programs is known to be 
^ high, and one might plausibly argue that 
many of the manuals either were not on 
hand when a change of prdgram directors 
took place, or the manuals simply had 
never come to the attention of the program 
director and were either lost or 
misappropriated since being received* 
Thus, it is Ibossible that a significant, 
but unknown, percentage of program 
directors simply had no opportunity to 
know whether the manuals had, ih fact, 
been received. 

These two manuals had somewhat different 
emphases from one another, with one being 



The programs which reported having 
received either of the two manuals were 
asked whether the manuals- t«d been used in 
conducting an actual evaluation, or in 
training of program staff in evaliiation 
methods. Slightly over one-fifth (21 
percent) of the programs which received 
the Manual for Drug Abuse Treatment 
Program Self-Evaluation reported having 
used it in an actual evaluation, while an 
additional 9 percent had not used it for 
an evaluation but had used it for training 
staff. A comparable percentage (19 
percent) of the programs which reported 
receiving Conducting Follow-up Research in 
Drug Treatment Programs used it in an 
actual evaluation, while 6 4>ercent used it 
for training program staff. 

These responses suggest that while some 
number of programs found the manual 
relevant and useful in actually conducting 
evaluation, the percentage doing so was 
considerably smaller than had been 
anticipated. One possible interpretation 
is that such materials alone are not 
highly effective and additional efforts in 
staff training and technical assistance, iq 



Table 8. Receipt, availability, and use of NIDA evaluation materials 

(N=341) 





Percent of Programs 




Item 


* Manual for Drug Abuse 
Treatment Proqram 
Self-Evaluation 


Conducting Followup 
Research in Drug 
Treatment Programs 


Both 


Received materials 


61*9 (N=341) 


44.9 (N=341) 


40.0 


Materials available at 
treatment site 


97*6 (N=211) 


96.5 (N=153) 




Use of materials 

a* in actual evaluation 
b. in training staff in 
in evaluation 


21.4 (N=211) 
9*4 (Ns211) 


> 

18.7 (Nrl53) 
6.5 (N=153) 





geared to in-trieatment evaluation, 
primarily, and the other, as its riame 
suggests, to followup evaluation* * While 
the percentage reporting having received 
either or both\)f these documents was 
considerably less than expected, the 
overwhelming majority of those programs 
which reported receiving one of the 
documents also reported having the 
document available for use at the program 
site. 



organizing and using data bases might 
V enhance the effective utilization of these 
manuals. Of course, it should also be 
kept in mind that some of the programs may 
not have been in operation at the time the 
manuals were disseminated, and a number of 
responding program directors may have come 
to the programs after the manuals were 
disseminated. TOrnover of staff in drug ' 
treatment programs is common and, there- 
fore, the number of respondents havir|<g no 
opportunity to know whether the manuals 
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had been received may be significant but 
unknown* 

Variables Related to Evaluation 

Several variables which were. thought to 
possibly be related to the performance of 
evaluation were examined in relation to 
the actual performance of evaluation. 
Using a dichotomous classification of 
programs on the basis of whether they were 
considered to have conducted evaluation 
during the year prior to this survey, 
selected variables were examined in terms 
of their statistical relationship to the 
conduct of evaluation. The size of the 
programs or clinic appeared to be 
statistically related to the performance 
of the evaluation (with a chi square value 
beyond the .01 level, and Cramer's V = 
•20) ♦ Thus, the larger programs, which, 
perhaps could be expected to have mofe 
resources for evaluation, appeared 
considerably more likely to conduct 
evaluation. Because of the relatively 
small numbers of methadone maintenance and 
drug free residential programs in the 
sample, as well as the large. proportion of 
"mixed modality" programs, no attempt was 
made to relate modality to the performance 
of evaluation. The use of urinalysis to 
monitor drug use during treatment appears 
to be related to the actual conduct of , 
evaluation. In some respects, the 
monitoring of drug use during treatment, 
especially using urine testing, may simply 
reflect the greater predisposition on the 
part of program management to use 
available tools for needed management 
information, includipg evaluation* In any 
event, the regular monitoring of drug use 
during treatment by urilnalysis was 
moderately related to the performance of , 
evaluation (chi square significant at .01 
level, Cramer's V = .2A). 

Also considered was the possibility that 
the receipt of one of the two evaluation 
manuals mentioned above might be related 
to the performance of evaluation. How- 
ever, the time interval of 2 years since 
dissemination and the high turnover among 
program staff would make interpretation of 
such a finding difficult. Therefore, this 
analysis was omitted. 

DISCUSSION 

The foregoing data indicates that 
substantial numbers of programs which had 
received NIDA funding at the time of the 
survey were found^ to have conducted 
evaluation duting" the previous yeafr. In 
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all, somewhat over two-fifths of these 
programs were conducting some sort of 
evaluation, and a substantial number of 
these were conducting more than one type 
of evaluation. The most prevalent type of 
evaluation was that most easily supported 
by existing client-oriented information 
systems— in-treatment evaluation. 
Followup evaluations were conducted by 
just under one-fifth of the programs 
surveyed. A relatively small number (7 
percent) of programs were found to have 
conducted process/cost evaluations that 
adhered to standard evaluation study 
criteria. 

Within programs, the evaluations tended to 
be local, relying largely on program staff 
or consultants, although substantial 
support by single State agencies was 
noted. The impetus for evaluation came 
from a variety of sources, both internal 
and external. While there appeared to be 
widespread recognition of the value of 
evaluation for internal planning and 
program administration, such external 
considerations as funding and licensure 
also appeared to be important. The use of 
systems already in place, such as CODAP, 
appeared to play a significant part in the 
evaluations conducted* Especially in the 
in-trreatment evaluation, programs appeared 
to place heavy reliance on CODAP admission 
and discharge data, as well as other data 
systems. For both in-treatment and 
followup evaluation, the more commonly 
accepted behavioral measures of treatment 
outcomes were, prominent in the structure 
of the evaluations, with considerable 
reliance on self-report of drug use, 
arrest, and productive activities 
including employment. Regarding 
PB^i^tment evaluation, monitoring of 
^^H|e through urinalysis appeared to be 
%IP^place. The most frequently used 
comparisons in analyzing data were 
comparisons of variables before treatment 
and after "treatment (in the case of 
in-treatment evaluation, of pretreatment, 
and at discharge), although considerable 
numbers of programs also made comparisons 
between subpopulations, clinics, and 
recently published CODAP data. ^ 

Programs conducting evaluation generally 
had well established procedures for 
disseminating evaluation results, with the 
majority reporting both to their own 
program staff and to outside organizations 
such as the single State agencies, NIDA, 
and other agencies. In addition, changes 
in program operations as a result of 
evaluations appeared to be widespread, 
with the most conspicuous changes being in 
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the direction of expanding program 
operations and improving services. 

In exandning factors which appeared to be 
statistically related to the performance 
of evaluation, clinic size appeared to be 
directly related, as did the regular 
collection of client drug use data during 
treatment. The evaluation manuals which 
were disseminated by NIDA 2 years prior to 
the study did not appear to be in 
widespread use, although about one-fifth 
of the programs reporting having received 
the manuals had actually used them in an 
evaluation.- 

Prospects for improving the performance of 
evaluation must necessarily be speculative 



at this point, since no attempt was made 
to assess the quality of evaluations being 
conducted by these programs §nd the 
responsibility for supporting evaluation 
has largely shifted to the State govern- * 
ments. Training and technical assistance 
offer avenues for improving the ability of 
programs to conduct. evaluation, although 
evaluation manuals may also be useful in 
conjunction with these efforts. The 
greatest perceived needs on the part of 
the programs surveyed were in the areas of 
staff training and data base organization; 
and, given a commitment on the part of 
treatment program management, these are 
opportunities which should be pursued. 
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